Overall and pairwise segregation tests based on nearest neighbor contingency tables

Multivariate interaction between two or more classes (or species) has important consequences in many fields and causes multivariate clustering patterns such as segregation or association. The spatial segregation occurs when members of a class tend to be found near members of the same class (i.e., near conspecifics) while spatial association occurs when members of a class tend to be found near members of the other class or classes. These patterns can be studied using a nearest neighbor contingency table (NNCT). The null hypothesis is randomness in the nearest neighbor (NN) structure, which may result from — among other patterns — random labeling (RL) or complete spatial randomness (CSR) of points from two or more classes (which is called the CSR independence, henceforth). In this article, we introduce new versions of overall and cell-specific tests based on NNCTs (i.e., NNCT-tests) and compare them with Dixon’s overall and cell-specific tests. These NNCT-tests provide information on the spatial interaction between the classes at small scales (i.e., around the average NN distances between the points). Overall tests are used to detect any deviation from the null case, while the cell-specific tests are post hoc pairwise spatial interaction tests that are applied when the overall test yields a significant result. We analyze the distributional properties of these tests; assess the finite sample performance of the tests by an extensive Monte Carlo simulation study. Furthermore, we show that the new NNCT-tests have better performance in terms of Type I error and power. We also illustrate these NNCT-tests on two real life data sets.